Skip to main content

Use Case – High crash count and high cpu usage on number of citrix servers in farm

Discovered​

UXM triggered alerts that high number of crashes was occurring on endpoint and that endpoints was experiencing higher than normal CPU usage.

 

Actions​

Technicians investigate which proceses that crashed and could see that it only occurred for msedgewebview2.exe on 6 of their 90 citrix servers.

mceclip0.png

Average CPU had increased from 20% to 70% since the 4th august due to msedgewebview2.exe crashing constantly, causing windows to launch a process dump through the process werfault.exe.

 

mceclip2.png

mceclip0.png

 

mceclip1.png

 

Solution​

Technicians investigated where msedgewebview2.exe was launched from and discovered that it was Outlook Desktop app that launched 6x msedgewebview2.exe processes when Calendar appointments was opened up by their Citrix users.

mceclip3.png

Only 6 out of 90 citrix servers had the Edge WebView2 embedded browser installed, because Office 365 was pushing it out automatically via the Office 365 package. Ref: https://docs.microsoft.com/en-us/deployoffice/webview2-install

CodeIntegrity events was also seen in the EventLog due to Citrix hooking into the msedgewebview2.exe process.

mceclip0.png

XenDesktop/XenApp 7.9 and later utilizes Kernel APC Hooking as a replacement of AppInit_DLLs used in previous versions. All Citrix Hooking (including MfApHook.dll and MfApHook64.dll) was disabled by creating the following registry value and the issue disappeared, ref: https://support.citrix.com/article/CTX107825/how-to-disable-citrix-api-hooks-on-a-perapplication-basis: 

Key: HKLM\SYSTEM\CurrentControlSet\services\CtxUvi
Value Name: UviProcessExcludes
Type: REG\_SZ
Value:msedgewebview2

The fix was pushed out after testing to the rest of the Citrix farm via GPO’s.